Chapter 1: Concepts and definitions

Goals

Understand conceptual and operational definitions

Know some strategies for clarifying conceptual definitions and writing conceptual definitions

Defining terms

Research Questions

  • Does religion play a role in civil wars?

  • What causes increasing income inequality in least developed countries?

  • Why has public opinion on same sex marriage liberalized so rapidly?

  • Is the U.S. becoming more polarized?

Not everyone is going to agree on what these terms mean!

Before we can get anywhere researching these questions, we need to define the thing we’re studying

Defining things: harder than it seems!

Reasonable definition of a chair

Chair

Terms

Conceptual Definition

A description of the concrete, measurable properties of a concept and the unit of analysis to which it applies

Unit of Analysis

The entity that that is being studied. For instance: individuals, governments, parties etc.

Operational Definition

A description of the instrument used to measure the concept

Crafting Conceptual Definitions: Unit of Analysis

  • What is the unit? What entity possesses the characteristic?
    • The U.S. and Canada are democracies (concept: democracy, unit: countries)
    • Lincoln, LBJ, and Trump are the three tallest U.S. presidents (concept: height, unit: presidents/people)
    • Labour is the UK’s main center-left political party (concept: party ideology/family, unit: political party)

One way to think of this: if you put your data into a spreadsheet, what does each “row” represent? The answer to that question is the unit of analysis.

Unit of analysis and the ecological fallacy

The ecological fallacy occurs when you attribute a characteristic of groups to individuals. For instance:

  • High-income states tend to vote for Democrats, but rich people tend to vote for Republicans.

  • Suicide rates in 19th century Europe were higher in Protestant countries compared to Catholic countries, but Protestant individuals were no more likely to die by suicide.

  • In the 1930 census, zip codes with more immigrants had higher literacy rates, but immigrant individuals had lower literacy rates.

All of these are fundamentally problems of ignoring the unit of analysis.

Crafting Conceptual Definitions: Key features

  • What are the essential features of the concept?
    • If there are cases or definitions everyone agrees on, what characteristics do they share?
    • Are certain qualities necessary or sufficient for a case to belong to a category?
    • What characteristics are most distinctive to those cases?
    • What is most helpful for clarifying edge-cases?
  • Are there multiple dimensions or just one? If two characteristics always occur together, you might only need to account for one of them!

Crafting Conceptual Definitions: transparency

We want to avoid being Potter Stewart:

The goal is cumulative knowledge, and a private definition can’t provide that.

Some steps to take:

  1. Identify the unit of analysis

  2. Make a list of important properties, clear examples and non-examples, and/or generally accepted definitions

  3. Remove items that aren’t measurable

  4. Reduce dimensions where possible (if a characteristic is shared by positive cases and negative cases, then it isn’t useful)

  5. Refine as needed

Conceptualizing Democracy

Some proposed features:

  1. Regular elections with meaningful alternatives

  2. Peaceful transfer of power

  3. Free expression

  4. A competitive media environment

  5. Autonomous political groups

  6. Rule of law

  7. Checks and balances

  8. Property rights

Conceptualizing Democracy

Unit of analysis: governments (usually national governments)

Case:

  • Definitely democracies: U.S., Uruguay, Taiwan, Japan, Botswana

  • Definitely autocracies: North Korea, Saudi Arabia, Russia

  • Edge cases: Hungary, Turkey, Tunisia

Features:

  • Voting and elections
  • Multiple parties
  • Meaningful civil liberties
  • Stability and monopoly on violence

This may need refinement: what about sham elections?

This may not be distinctive: there are stable autocracies on the list

Operational Definitions

Operationalization

Even where we agree on a definition, we will need to measure a concept and there is often slippage here

Considerations: Parsimony

In many cases, we trade some truth for simplicity. All else equal, a simpler definition is better than a more complicated one.

Considerations: Parsimony

This is a very accurate rendering of the DC metro area

But this is probably more useful for getting around.

Considerations: Parsimony

Similarly: we might consider “democracy” to be a highly complex multidimensional concept

mindmap
  root((Democracy))
    Liberal Rights
        Free Speech
        Free Association
        Rule of law
    Deliberation
        Respectful Dialogue
        Adversarial Press
        Broad Participation
    Egalitarianism
        Equal Rights
        Equitable access to resources
        Diverse representation
    Majoritarianism
        Fair elections
        Peaceful transfer of power
        Meaningful alternatives

Considerations: Parsimony

…but we might still prefer an operational definition that relies on a subset of concepts that are easy to measure.

mindmap
  root((Democracy))
    Majoritarianism
        Fair elections
        Peaceful transfer of power
        Meaningful alternatives

Operationalizing Democracy

Dictatorship and Democracy (Alvarez, Cheibub, Limongi, & Przeworski 1996)

Dichotomous measure. Democracies have the following:

  • Have popular elections to fill seats

  • Have more than one party

  • The incumbent sometimes loses

Varieties of Democracy (V-DEM) Project (link)

Democracy along multiple dimensions. Scores are determined by:

  • Polling a large group of area experts on a range of democratic dimensions (electoral, deliberative, liberal etc.)

  • Using an algorithm to aggregate those responses into scores along each dimension.

Considerations: Reliability

Reliability refers to how consistently the same measurement instrument produces the same result

Measuring reliability

  • Test-Retest method: give the same test to the same group twice. Measure the correlation between answers.

Measuring reliability

Zaller, J., & Feldman, S. (1992). A simple theory of the survey response: Answering questions versus revealing preferences. American journal of political science, 579-616.

Measuring reliability

  • Split half: for a measurement using a multi-item scale, we can split results and use half the questions to predict the remaining half.

    • If the scale questions all measure the same “thing” then you would expect the halves to resemble each other.

Considerations: Validity

  • Validity refers to whether an operational definition correctly measures the concept we’re interested in.

  • A challenge to validity might come from cases that seem mis-categorized in our definition. For example: the minimalist definition of democracy suggests that Japan was authoritarian for much of the latter half of the 20th century.

Considerations: Validity and Reliability

  • Measures could be reliable but invalid: your astrological sign is a reliable measure of your political views because it doesn’t change, but it also isn’t valid because it has no real correspondence to your actual politics.

  • Measures could be valid but unreliable: an exam with a single well-written question might be a valid measure of a student’s understanding of material, but it will probably have more random variability compared to an exam with multiple poorly-written questions.

Measurement Error

  • Random Error
    • Radio interference
    • Data entry errors
    • Random non-response
  • Systematic Error
    • Hawthorne effect (people behave differently when they know they’re being observed)
    • Social desirability bias (people don’t want to admit to doing bad stuff, even on an anonymous survey
    • Systematic non-response (surveys) or missing data (everything)

Measurement Error

  • Measurement error is a source of noise, systematic measurement error is a source of bias

  • We’ll generally find that random noise is much easier to deal with than bias because we can simply collect more data. Significant bias, on the other hand, presents profound challenges to research because we can only correct it if we can measure it

Measuring Party ID

Consider measuring party identification:

Instrument Problem
Registration Excludes new/non-voters, people in states without partisan registration, and people who register one way and vote another
Policy views Complicated to measure, and a surprising number of people don’t have consistent policy views!
Voting Behavior Excludes new/non-voters, may not be consistent even in a single election, people may exaggerate or forget voting behavior.
Self Description* Subjective, far more self-described “independents” than people who consistently vote that way

Party ID

  • Sources like the American National Election Study rely on self-description to measure party ID, but each respondent receives two questions instead of one.

Party ID

Roughly a third of the electorate describe themselves as “independent”

Party ID

When pressed, many independents will say they lean toward a party.

Party ID

  • Based on voting behavior, “leaners” are more similar to partisans.
  • So we commonly measure party ID with two questions.

Party ID

  • The ANES method for measuring party ID is hardly the only option we have, but it has some nice features:
    • It’s relatively reliable and survey respondents understand it
    • It’s close to the conceptual definition of party ID as a sort of self-identity or group affinity.
    • It’s transparent and widely used
    • It limits the influence of social desirability bias by giving respondents lots of options
    • It’s linked to an observed behavior: vote choice

Key points

  • Asking social science questions requires us to define complex ideas and measure them. And we can lose a lot in that process.

  • There are some wrong answers, but there is generally no single correct answer. We generally have to make trade-offs between things we might value like:

    • Parsimony
    • Reliability
    • Validity
    • Probability of random or systematic errors

  • We can’t really achieve perfection, but where there is disagreement, we want to be transparent about methods and limitations.

Reminders

  • Install R and R-Studio for Friday discussion

  • Start thinking about questions for pilot design